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1 Introduction 

A natural problem is to understand what a typical element of the finite general linear group GL{n, q) 
"looks like". Many of the interesting properties of a random matrix depend only on its conjugacy 
class. The following list of questions one could ask are of this type: 

1. How many Jordan blocks are there in the rational canonical form of a random matrix? 

2. What is the distribution of the order of a random matrix? 

3. What is the probability that the characteristic polynomial of a random matrix has no repeated 
factors? 

4. What is the probability that the characteristic polynomial of a random matrix is equal to its 
minimal polynomial? 

5. What is the probability that a random matrix is semisimple (i.e. diagonalizable over the 
algebraic closure F q of the field of q elements)? 

As Section || will indicate, answers to these questions have applications to the study of random 
number generators, to the analysis of algorithms in computational group theory, and to other parts 
of group theory. Section |2| describes a unified approach to answering such probability questions 
using cycle index generating functions. As an example of its power, it is proved independently in 
flFl|l and |W2| that the n — > oo limit of answer to Question 3 is (1 — ^)/(l + There is (at 
present) no other method for deriving this result and generating functions give effective bounds on 
the convergence rate to the limit. Extensions of the cycle index method to the set of all matrices 
and to other finite classical groups are sketched. 

Section ||| gives a purely probabilistic picture of what the conjugacy class of a random element 
of GL(n, q) looks like. The main object of study is a probability measure McL,u,q on the set of all 
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partitions of all natural numbers. This measure is connected with the Hall-Littlewood symmetric 
functions. Exploiting this connection leads to several methods for growing random partitions 
distributed as McL,u,q and gives insightful probabilistic proofs of group theoretic results. We hope 
to convince the reader that the interplay between probability and symmetric functions is beautiful 
and useful. A method is given for sampling from Mgl,u,q conditioned to live on partitions of a 
fixed size (which amounts to studying Jordan form of unipotent elements) and for sampling from 
a g-analog of Plancherel measure (which is related to the longest increasing subsequence problem 
of random permutations). 

Section || goes on to describe a probabilistic approach to McL,u,q using Markov chains. This 
connection is quite surprising, and we indicate how it leads to a simple and motivated proof of the 
Rogers- Ramanuj an identities. The measure McL,u,q has analogs for the finite unitary, symplectic, 
and orthogonal groups. As this is somewhat technical these results are omitted and pointers to 
the literature are given. However we remark now that while the analogs of the symmetric function 
theory viewpoint are unclear for the finite symplectic and orthogonal groups, the connections with 
Markov chains carry over. Thus there is a coherent probabilistic picture of the conjugacy classes 
of the finite classical groups. 

Section || surveys probabilistic aspects of conjugacy classes in T(n, q), the group of n x n upper 
triangular matrices over the field F q with l's along the main diagonal. Actually a simpler object 
is studied, namely the Jordan form of randomly chosen elements of T(n, q). From work of Borodin 
and Kirillov, one can sample from the corresponding measures on partitions. We link their results 
with symmetric function theory and potential theory on Bratteli diagrams. 

The field surveyed in this article is young and evolving. The applications to computational group 
theory call for extensions of probability estimates discussed in Section ^ to maximal subgroups of 
finite classical groups. It would be marvellous if the program surveyed here carries over; this 
happens for the finite affine groups |F£]. The first step is understanding conjugacy classes and 
partial results can be found in the thesis |Mu]. 

We close with a final motivation for the study of conjugacy classes of random matrices over 
finite fields. The past few years have seen an explosion of interest in eigenvalues of random matrices 
from compact Lie groups. For the unitary group U(n, C) over the complex numbers, two matrices 
are in the same conjugacy class if and only if they have the same set of eigenvalues. Hence, at 
least in this case, which is related to the zeroes of the Riemann zeta function [KeaSnJ, the study of 
eigenvalues is the same as the study of conjugacy classes. 



As complements to this article, the reader may enjoy the surveys |Sh2fl , | Py 1 1 , [ Py2 1 . These 
articles discuss probabilistic and enumerative questions in group theory and have essentially no 
overlap with the program surveyed here. 



2 Cycle Index Techniques 

Before describing cycle index techniques for the finite classical groups, we mention that the cycle 
index techniques here are modelled on similar techniques for the study of conjugacy class functions 
on the symmetric groups. For a permutation ir, let n^n) be the number of length i cycles of ir. 
The cycle index of a subgroup G of S n is defined as 

11 TT&Gi>l 

and is called a cycle index because it stores information about the cycle structure of elements of G. 



Applications of the cycle index to graph theory and chemical compounds are exposited in [PoRe] 
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It is standard to refer to the generating function 



n>l 



as the cycle index or cycle index generating function of the symmetric groups. Prom the fact that 
there are 



elements in S n with n{ cycles of length i, one deduces Polya's result that this 

. This allows one to study conjugacy class functions of 



generating function is equal to ]lm>i e ™ 
random permutations (e.g. number of fixed points, number of cycles, the order of a permutation, 
length of the longest cycle) by generating functions. We refer the reader to [ [Kg 1 ] for results in this 



direction using analysis and to [ShLl| for results about cycle structure proved by a probabilistic 
interpretation of the cycle index generating function. 

Subsection |27j] reviews the conjugacy classes of GL(n,q) and then discusses cycles indices for 
GL(n, q) and Mat(n, q), the set of all n x n matrices with entries in the field of q elements. Subsec- 
tion ^2] describes applications of cycle index techniques. Subsection 22 discusses generalizations 
of cycle indices to the finite classical groups. 

It is necessary to recall some standard notation. Let A be a partition of some non-negative 
integer |A| into integer parts Ai > A2 > • • • > 0. We will also write A h n if A is a partition of 
n. Let m,i{\) be the number of parts of A of size i, and let A' be the partition dual to A in the 

sense that A- = nii(X) + mj + i(A) -| . Let n(A) be the quantity X)i>i(* 

(l-f)---(l-f)- 

2.1 The General Linear Groups 



l)Aj and let (^)j denote 



To begin we follow Kung | Kun| in defining a cycle index for GL(n,q). First it is necessary to 
understand the conjugacy classes of GL(n,q). As is explained in Chapter 6 of the textbook [ Her ], 
an element a G GL(n,q) has its conjugacy class determined by its rational canonical form. This 
form corresponds to the following combinatorial data. To each monic non-constant irreducible 
polynomial <fi over F q , associate a partition (perhaps the trivial partition) A^ of some non-negative 
integer |A<a|. Let deg(4>) denote the degree of (p. The only restrictions necessary for this data to 
represent a conjugacy class are that |A 2 | = and J2d> \^(t>\deg(4>) = n. 

An explicit representative of this conjugacy class may be given as follows. Define the companion 



matrix C{4>) of a polynomial 4>(z) = z 9 ^> + a deg ^ 



( 




V 





-a 



1 






y deg(<t>)-l 






- a deg(4>)-l J 



+ • • • + ot\Z + ao to be: 



Let <f>±, ■ ■ ■ , (f>k be the polynomials such that | A^ | > 0. Denote the parts of A^ by A^i > A^^ > • • •■ 
Then a matrix corresponding to the above conjugacy class data is 



/ Ri 








\ 





R2 








V 








R k ) 



where R; is the matrix 
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... J 



For example, the identity matrix has \ z -\ equal to (l n ) and all other equal to the emptyset. 
An elementary transvection with a / in the (1,2) position, ones on the diagonal and zeros 
elsewhere has A z _i equal to (2, l n ~ 2 ) and all other equal to the emptyset. For a given matrix 
only finitely many A^ are non-empty. Many algebraic properties of a matrix can be stated in 
terms of the data parameterizing its conjugacy class. For instance the characteristic polynomial 
of a £ GL(n,q) is equal to Yls^ an d the minimal polynomial of a is equal to n^'^' 1 ^'- 
Furthermore a is semisimple (diagonalizable over the algebraic closure F q ) precisely when all A,* (a) 
have largest part at most 1. 

To define the cycle index for Z GL ^ nq ^, let be variables corresponding to pairs of polynomials 
and partitions. Define 



1 



J GL{n,q) 



n 



1 K ,q)l aeGL(n,q) fli:|A («)|>O 



Note that the coefficient of a monomial is the probability of belonging to the corresponding conju- 
gacy class, and is therefore equal to one over the order of the centralizer of a representative. It is 
well known (e.g. easily deduced from page 181 of [ Macf ) that one over the order of the centralizer 
of conjugacy class of GL(n,q) corresponding to the data {A^} is 



1/ >_l\ q deg{4>) )tHi(A^) 



The formulas given for conjugacy class size in [Kun| and [Stl] are written in different form; for the 
reader's benefit they have been expressed here in the form most useful to us. It follows that 



1 + E Z GL(n, q) U n = II 
n=l <t>^z 



u n-deg{4,) 

n>l\\-n Q ^ l 1 li>l \ q deg 



This is called the cycle index generating function. 

Let Mat(n,q) be the set of all n x n matrices over the field F q . Define 



Z 



Mat{n,q) ~ ifir ( _\| E II x </-,A^(a)- 

1 V hHn a£Mat(n,q) </.:|A <A (a)[>0 



Analogous arguments [Stl] show that 



1 + E 



n 



n=l 



n>lAhn q 9WZj ' 1 J Ui>l 



This will be used in Subsection Note that the denominator in Z Mat ^ nq ^ is \GL(n,q)\, not 
\Mat(n, q)\, since the formula follows from a formula for the size of the orbits of GL(n,q) acting 
on Mat(n, q) by conjugation. This makes no essential difference for applications. 
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2.2 Applications 



This subsection describes applications of cycle indices. The first example is treated in detail and 
results for the other examples are sketched. 

Example 1: Cyclic and Separable Matrices 

Recall that a matrix a £ Mat(n, q) operating on a vector space V is called cyclic if there is a 
vector vq £ V such that vq, vqo, vqo 2 , ■ ■ ■ span V. As is explained in NP2| ], this is equivalent to the 



condition that the characteristic and minimal polynomials of a are equal. 



The need to estimate the proportion of cyclic matrices arose from [NP1| in connection with 
analyzing the running time of an algorithm for deciding whether or not the group generated by a 
given set of matrices in GL(n,q) contains the special linear group SL(n,q). Cyclic matrices also 
arise in recent efforts to improve upon the MeatAxe algorithm for computing modular characters 



| NP4 | and in Example 8 below. John Thompson has asked if every matrix is the product of a 
cyclic matrix and a permutation matrix, suggesting that the answer could have applications to 
finite projective planes. 



Letting CM(n,q) be the proportion of cyclic elements of Mat(n,q), the paper [NP2] proves that 



1 i / x 1 

< 1 - c M {n,q) < 



The cycle index approach is also informative, yielding a formula for the n — > oo limit of Cm(^, q), 
denoted by cjvf (oo, g), together with convergence rates. For the argument two lemmas are useful, 
as is some notation. Let Nd(q) be the number of degree d irreducible polynomials over the field 
F q . In all that follows <p will denote a monic irreducible polynomial over F q . Given a power series 
f(u), let [u n ]f(u) denote the coefficient of u n in f(u). 



Lemma 1 



u deg(4>) 
qdeg{<j>) ■ 



U 



Proof: Expanding d eg u) as a geometric series and using unique factorization in -F 9 [x], one sees 

that the coefficient of u d in the reciprocal of the left hand side is \ times the number of monic 
polynomials of degree d, hence 1. Comparing with the reciprocal of the right hand side completes 
the proof. □ 

Lemma 2 If the Taylor series of f around converges at u = 1, then 

lim n ^ 00 [u n }P^=f(l). 
1 — u 

Proof: Write the Taylor expansion f{u) = J2™=o a nU n ■ Then observe that [u n ]j^l = YA=o a i- u 
Theorem |l] calculates cm {oo, q). 

Theorem 1 (W\hW$l) 

1 00 1 

CM(0C,,) = (1- -,)m- V r) 
" r=3 " 
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Proof: Recall that a is cyclic precisely when its characteristic polynomial and minimal polynomials 
are equal. From Subsection |2.l| , these polynomials are equal when all A^ have at most one part. In 
the cycle index for Mat(n, q) set x<f, t \ = 1 if A has at most 1 part and = otherwise. It follows 
that 

. , \GL(n,q)\. niTT/1 ^ u rdeg&) 
CM(n,q)= ^ [u ] LL(1 + ^ g(j ._ 1)degW (gdegW _ 1} )■ 

By Lemma [l] this equation can be rewritten as 
c M (n,q) = J V^K 1 



q n l-u 

nl-, I u de 3 (4>) , 
,.. ,, <A + gdegW^gdegW^ ) 

^2 l U \ \ 

q n 1 — u 



|GL(n,g)L„ i n d >i(l + ^)^ 



g n 1 - u 

Recall that a product Il^i(l + a n) converges absolutely if the series J2 n >i \ a n\ converges. Thus 
using the crude bound N^q) < q d 

na + - ) Nd{q) 

4 A i ^ d -i) J 

is analytic in a disc of radius greater than 1. Lemma |2] implies that 



q n ' 1 — u 

17(1- — ) TTfl + - ) Ndig) 



Applying Lemma |T] (with u = -, u = and then u = gives that 

00 1 1 11 

r=3 y d>l ^ ^ ' y y 

= nu-^na-i)^ 



r=3 «' d>l 9" 

1 °° 1 

(1 -?>np- ? >- 



□ 



The next challenge is to bound the convergence rate of CM(n, q) to Coo(n, q). Wall | W2 | found a 
strikingly simple way of doing this by relating the cycle index of cyclic matrices to the cycle index 
of the set of matrices whose characteristic polynomial is squarefree (these matrices are termed 
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separable in | NP2(| ). To state the result, let SM{n,q) be the probability that an n x n matrix is 



separable. Next let Cm(u,q) arid Sm(u,q) be the generating functions defined as 



u n q n2 



C M (u, q) = l + J2 77777 rrC M (n, q) 

^ \GL(n,q)\ 



SM(u, q ) = l + ^ m — ri s M (n,q). 



Lemma 3 (\WW) 



(1 - u)C M (u, q) = S M (u/q, q). 
Proof: The proof of Theorem || shows that 



U ^N d {q) 



(l-u)C M (u, q ) = l[(l+ ) 

d>l ^ W ' 



A matrix is separable if and only if all have size or 1. Hence 

d 



S M (u,q) = IK 1 + 7/37 



u 

(1 + 7 

d>l 

The result follows. □ 



N d (q) 

q d 



Corollary 1 (\W%j) 

< \c M (n,q) - c M (00, q)\ < 



q n+i(i _ i/ q ) 

Proof: Taking coefficients of u n+1 on both sides of Lemma |3] gives the relation 

s M {n + l,q) - c M (n,q) 



c M {n + l,q) - c M {n,q) 



q n+l 



Since < |sm("- + — CM(n,q)\ < 1 for all n, it follows that 



\c M (n,q) - c M (oo,q)\ < \c M (i + l,q)- c M (i,q)\ < 



q i+l ■ 

i=n 3 



as desired. □ 
Remarks: 

1. As mentioned in the introduction, an argument similar to that of Theorem p] shows that the 
n — > 00 probability that an element of GL(n,q) is cyclic is (1 — ^-)/(l + For large q 
this goes like 1-l/q 3 . The reason for this is a result of Steinberg [|5tei| stating that the set 
of non-regular elements in an algebraic group has co-dimension 3. In type A, regular (i.e. 
centralizer of minimum dimension) and cyclic elements coincide, but not always. For more 
discussion on this point, see | NP2fl , []FNP]. 
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2. The generating functions Sm(u,q) and Cm{u, q) have intriguing analytical properties. It is 
proved in | W2 | that 



S M (u,q) 



u d (u d -l) )N J a ) 
q d (q d -l) > 



1 



Thus Sm( u ,q) has a pole at 1 and Sm{ u iQ) ~ can be analytically extended to the circle 
of radius q. Analogous properties hold for Cm(u, q) by means of Lemma ||. 



3. The limits SM(oo,q) and SQL(oo,q) are in [F1|, [W2(| . Bounding the rate of convergence of 
SM(n, q) to sm (oo, q) leads to interesting number theory. Let p(d) be the number of partitions 



of d and let P2(d) = J2i=oP(d)- It is proved in |W2[ that 
s M (n,q)q n ' 



\GL(n,q) 



* 1 A a _i_ 97 9 

1|< E (P2(d)+^-2)) (7 - d <-(4^ r )(- ( z)-". 



c(=n+l 



3 v 2q - 3 



4. Lehrer [Leh| expresses sm(it-, q) and SGL(n, q) as inner products of characters in the symmetric 
group and proves a stability result about their expansions in power of q^ 1 . See also | W2 | and 



|LehSe|. 



5. The results of [F2] and [|W2[ surveyed above are extended to the finite classical groups in 



[ FNP |. The paper [F1J] gives (intractable) formulas for the chance of being separable in 
groups such as SL(n,q) (i.e. semisimple and simply connected). 

Example 2: Eigenvalue free matrices 

The paper [NP3] studies eigenvalues free matrices (i.e. matrices without fixed lines) over finite 
fields as a step in obtaining estimates of cyclic probabilities in orthogonal groups (np|. it IS 
interesting that the study of eigenvalue free matrices was one of the motivations for the original 
papers fKunfl , [ Stl | , the latter of which proves that the n, q — > oo limit of the chance that an element 
of GL(n,q) has no eigenvalues is -. 

The n — » oo probability that a random element of S n has no fixed points is also -. This 
is not coincidence; in general the q — * oo limit of the chance that the characteristic polynomial 
of a random element of Mat(n, q) factors into n>i degree i irreducible factors is the same as the 
probability that an element of S n factors into ra, cycles of degree i. This is proved at the end of 
ptlfl and is extended to finite Lie groups in |F1| using the combinatorics of maximal tori. There 
is another interesting line of argument which should be mentioned. It is easy to see from the cycle 
index that the factorization type of the characteristic polynomial of a random element of Mat(n, q) 
and the factorization type of a random degree n polynomial over F q have the same distribution as 
q — * oo. Now the factorization type of a random degree n polynomial over F q has same distribution 
as the cycle type of a random permutation distributed as a g-shuffle on n cards [DiaMcPi], and as 
q — > oo a (/-shuffle converges to a random permutation. The connection of Lie theory with card 



shuffling may seem adhoc, but is really the tip of a deep iceberg [F1C]. 
Example 3: Characteristic polynomials 

The previous example is a special case of the problem of studying the degrees of the factors of 
the characteristic polynomial of a random matrix. Many results in this direction (all proved used 



cycle indices) can be found in Stong's paper | Stl ] . Hansen and Schmutz [ HSchm ] use cycle index 
manipulations to prove that if one ignores factors of small degree, then the factorization type of 
the characteristic polynomial of a random element of GL(n, q) is close to the factorization type of a 
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random degree n polynomial over F q . More precisely, let A n> i be the set of sequences (ety+i, • • • , a n ) 
where oii is the number of degree i factors of a random polynomial chosen from some measure. Let 
Q { n ] be the measure on polynomials arising from characteristic polynomials of random elements of 

(2) 

GL(n, q) and let Q n be the measure arising from choosing a degree n polynomial over F q uniformly 
at random. They prove 

Theorem 2 ( frHSchn^j ) There exists constants c\,C2 such that for all I with c\log(n) < I < n and 
B C N n ~ l , 

\Q£\A n (B))-QW(A n (B))\<c 2 /l. 

The final section of their paper uses this principle to prove results about characteristic polynomials 
of random matrices using known results about random polynomials. A useful reference on the 



distribution of degrees of random polynomials over finite fields is [ArBarT]. 
Example 4: Generating transvections 

Recall that the motivation behind Example 1 was a group recognition problem, i.e. trying to 
determine whether or not the group generated by a given set X of matrices in GL(n, q) contains the 
special general linear group SL(n, q). However the problem still remains of making the recognition 
algorithm constructive. For instance if the group generated by X is GL(n,q) it would be desirable 
to write any element of GL(n, q) word in X. 



The paper [CeLg] proposes such a constructive recognition algorithm. An essential step involves 
constructing a transvection, that is a non-identity element of SL(n, q) which has an n—l dimensional 
fixed space. This in turn is done in two steps. First, find an element a of GL(n, q) conjugate to 
diag(C((z — t) 2 ),R) where C is the companion matrix as in Subsection |2.1| and R is semisimple 
without r as an eigenvalue. Second, one checks that raising a to the least common multiple of the 
orders of r and R gives a transvection. 

Thus it necessary to bound the number of feasible a in the first step. Such a have conjugacy 
class data A 2 _ r = (2) and all other A^ have largest part at most 1. The cycle index approach gives 
bounds improving on those in flCeLg | ; see [FNP] for the details. 



Example 5: Semisimple matrices 

A fundamental problem in computational group theory is to construct an element of order p. 
Given a group element g with order a multiple of p, this can be done by raising g to an appropriate 



power. It is proved in [tsKanSp] that if G is a permutation group of degree n with order divisible 
by p, then the probability that a random element of G has order divisible by p is at least K 

Their proof reduces the assertion to simple groups and then uses the classification of simple 
groups. Let us consider the group GL(n,q), which is close enough to simple to be useful for the 
applications at hand. When p is the characteristic of the field of definition of GL(n, q), an element 
has order prime to p precisely when it is semisimple. Thus the problem is to study the probability 
that an element of GL(n, q) is semisimple. The paper [ GuLubfl shows that if G is a simple Chevalley 
group, then the probability of not being semisimple is at most 3/(q — 1) + 2/(q — l) 2 and thus at 
most c/q for some constant c as conjectured by Kantor. 

As mentioned earlier, a matrix a is semisimple if and only if all A^(a) have largest part size 
at most 1. Stong |Stlj| used cycle indices to obtain crude asymptotic bounds for the probability 
that an element of GL(n, q) is semisimple. The thesis |F1| used the Rogers- Ramanuj an identities 
to prove that the n — > oo probability that an element of GL(n, q) is semisimple is 

oo fi ]_\ 



(i-f) 

r = 0,±2(mod 5) 



10 



The paper [FNP] gives effective bounds for finite n. 
Example 6: Order of a matrix 

A natural problem is to study the order of a random matrix. This has been done in [St2] 
and [Schm]; see also the remarks in Subsection 3.3 and the very preliminary calculations for other 
classical groups in [Fl|. Shalev |Shlf uses facts about the distribution of the order of a random 
matrix together with Aschbacher's study of maximal subgroups of classical groups [As] as key tools 
in studying the probability that a random element of GL(n, q) belongs to an irreducible subgroup 
of GL(n, q) that does not contain SL(n, q). As explained in [ |5hl|| this has a number of appications; 
for instance it leads to a proof that if x is any non-trivial element of PSL(n, q) then the probability 
that x and a randomly chosen element y generate PSL(n,q) tends to 1 as q — > oo. Shalev |Shl| 
asks for extensions of these results to other finite classical groups. 

It is also useful to count elements of given orders (e.g. 2 or 3) in classical groups and their 



maximal subgroups. The recent paper [CTY] uses cycle indices to perform such enumerations. 
One motivation for such enumerations is the study of finite simple quotients of PSL(2, Z); a group 
G is a quotient of PSL(2, Z) if and only if G =< x,y > with x 2 = y 3 = 1. For further discussion, 
see [Hhl 



Example 7: Random number generators 

We follow | Mar 1,| Mar Ts| in indicating the relevance of random matrix theory to the study of 
random number generators. Suppose one wants to test a mechanism for generating a random 
integer between and 2 33 — 1. In base 2 these are length 33 binary vectors. Generating say n of 
these and listing them gives an n x 33 matrix. If the random generator were perfect, the arising 
matrix would be random. One could choose a statistic such as the rank of a matrix and compare 
the generation method with theory. They report that shift-register generators will fail such tests 
but that congruential generators usually pass. It would be interesting to see how various random 
number generators perform when tested using other conjugacy class functions of random matrices. 

Diaconis and Graham [ piaGr ] analyze random walks of the form X n = + e n where Xi 

is a length d — 1 vector, A is an element of GL(n, 2), and e n is a random vector of disturbance 
terms. For more general A (in GL(n,q)) this includes the problem of running a psuedo-random 
number generator with recurrence Y n = aiY n _i + • • • + a^Y^d + e n with Yi £ F q . They show that 
the rational canonical form of A is related in a subtle way to the convergence rate of the walk. It 
would be interesting to understand what happens when A is a random matrix. 

Example 8: Product replacement algorithm 

In recent years finite group theory has become much more computational. Given a generating 
set S of a finite group G, it is natural to seek random elements of G. One approach, implemented in 
the computer systems GAP and MAGMA, is the product replacement algorithm [CeLgMuNiOb]. 
Fixing G and some k, one performs a random walk on fc-tuples (g\, ■ ■ ■ ,gk) of elements of G which 
generate the group. The walk proceeds by picking an ordered pair with 1 < i ^ j < n 

uniformly at random and applying one of the following four operations with equal probability: 



R t,3 ■ 



,9i 



,9k) 



(91 



, 9i • 9i 



,9k) 



L % ■ (9i, ■ ■ ■ , 9i, ■ ■ ■ , 9k) >-> (91, ■ ■ ■ , gf ■ 9i, ■ ■ ■ , 9k)- 

These moves map generating /c-tuples to generating /c-tuples. One starts from any generating k- 
tuple, applies the algorithm for r steps, and then outputs a random entry of the resulting fc-tuple 
(i.e. a group element). 

The product replacement algorithm has superb practical performance (often converging more 
rapidly than random walk on the Cayley graph), in spite of the theoretical defects that a random 
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entry of a random generating /c-tuple does not have the same distribution as a random element of 
G, and that the convergence rate of the chain on fc-tuples to its stationary distribution is unknown. 



The paper [CeLgMuNiOb|, aware of these issues, tests the algorithm against theory, using conjugacy 
class statistics such as the order of an element, the number of factors of the characteristic polynomial 
of a random matrix, the degree of the largest irreducible factor of the characteristic polynomial 
of a random matrix, and the proportion of cyclic matrices in the finite classical groups. In short, 
understanding properties of random matrices is crucial to their analysis. 

A recent effort to understand the performance of the product replacement algorithm uses Kazh- 
dan's property (T) from the representation theory of Lie groups |LubPa |; the paper [Pa] is a useful 
survey. Much remains to be done. 

Example 9: Running times of algorithms 

One of the main approaches to computing determinants and permanents of integer matrices 



involves doing the computations for reductions mod prime powers. Section 4.6.4 of [Kn| gives a 
detailed discussion with references to literature on upper bounds of running times. If one believes 
that typical matrices one encounters in the real world are like random matrices, this motivates 
studying random matrices over finite fields. In fact von Neumann's interest in eigenvalues of random 
matrices with independent normal entries arose from the same heuristic applied to questions in 
numerical analysis (the introduction of jj|] gives further discussion of this point). 

Examples of algorithms in which properties of random matrices were really needed to bound 
running times include recognizing when a group generated by a set of matrices contains SL(n,q) 



|NP1] and the Meat Axe algorithm for computing modular characters | NP4 |. 
Example 10: Isometry classes of linear codes 



Fripertinger Fripl] , [Frip2] considers cycle indices (in the permutation sense) of matrix groups 
acting on lines. His interest was in understanding properties of random isometry classes of linear 
codes-a harder problem than understanding random linear codes. The cycle indices he obtains 
seem quite intractable for theorem proving, but are useful in conjunction with computers. He also 
gives references to the switching function literature. 

Curiously, understanding the permutation action of random matrices of lines comes up in an- 



other context. Wieand [ Wi] has shown that the eigenvalues of random permutation matrices possess 
a structure similar to the eigenvalues of matrices from compact Lie groups. Persi Diaconis has sug- 
gested that the eigenvalues of high dimensional representations of finite groups of Lie type (such as 



the permutation action on lines) may possess similar structure; see [F5] for more in this direction 



2.3 Generalization to the Classical Groups 

This subsection will focus on the finite unitary groups, with remarks about symplectic and or- 



thogonal groups at the end. These cycle indices were derived in [Fl| ,[F2] and were applied to the 
problem of estimating proportions of cyclic, separable, and semisimple matrices (these terms were 
defined in Subsection |2.2[) in |FNP[] . 

The unitary group U(n,q) can be defined as the subgroup of GL(n,q 2 ) preserving a non- 
degenerate skew-linear form. Recall that a skew-linear form on an n dimensional vector space V 
over F q 2 is a bilinear map <, >: V x V — ► F q 2 such that < x, y >=< y,x > q (raising to the qth 
power is an involution in a field of order q 2 ). One such form is given by < x, y >= Ya=i x iV\- Any 
two non-degenerate skew-linear forms are equivalent, so that U(n, q) is unique up to isomorphism. 

Wall Wl ] parametrized the conjugacy classes of the finite unitary groups and computed their 
sizes. To describe his result, an involution on polynomials with non-zero constant term is needed. 
Given a polynomial (f> with coefficients in F q 2 and non vanishing constant term, define a polynomial 

4> by: 
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( a deg(<t>)~l \ q , / 1 \c, 



where 4> q raises each coefficient of </> to the gth power. Writing this out, a polynomial cft(z) = 

1 + - ■ ■+a 1 z+a with a + is sent to 4>{z) = z de ^ +(|)% & 9W-i + . . .+ 

. , q . An element a E Uin.q) associates to each monic, non-constant, irreducible 

polynomial over i^ g 2 a partition A^, of some non-negative integer | A^ | by means of rational canonical 
form. The restrictions necessary for the data A^ to represent a conjugacy class are that |A 2 | = 0, 
A<£ = A^, and that Wt>\deg(</>) = n. 

Using formulas for conjugacy class sizes from [Wl] together with some combinatorial manipula- 



tions, the following unitary group cycle index generating function was derived in [Fl]. The products 
in the theorem are as always over monic irreducible polynomials. 

Theorem 3 



e n x *m°) 



n=l \ U ( n 'Q)\ ae!7(n,g)^:|A^(a)|>0 



n 



1 + E E X ^ 

n>l Ahn 



( Q) ^ lli>l ^ ( _g)d e9 (« J 



mi (A) 



n 



1 + E E ^A^, A - 

n>lAhra 1 



2n-deg{<t>) 



tdeg^-^SK) 2 



i>l 



1 



,2deg(4,) 



rrii(A) . 



One interesting theoretical result concerning the cycle index of U(n, q) is the following functional 
equation. Letting Ccl(u, q) and Cu(u, q) be the cycle index generating functions for cyclic matrices 
in the general linear and unitary groups respectively, the functional equation states that 

CGL(u,q)Cu{-u, -q) = C GL (u 2 ,q 2 ). 



The paper | {FNP | proves that this relation holds whenever the condition on the partitions A^ is 
independent of the polynomial (f>. In the current example, a matrix is cyclic if and only if all A<^ 
have at most one row. This condition is independent of 4>. 

Cycle indices for the symplectic and orthogonal groups are a bit trickier to establish from Wall's 
formulas. To the treatment in |F1|,[F2] we add a remark which should be very helpful to anyone 
trying to use those cycle indices. The paper [F2] only wrote out an explicit formula for the cycle 
index for the sum of +,— type orthogonal groups. To solve for an individual orthogonal group, 
it is necessary to average that formula with a formula for the difference of +,— type orthogonal 
groups (this procedure is carried out in a special case in [ FNFfl ). In general, the formula for the 
difference of orthogonal groups is obtained from the formula for the sum of orthogonal groups as 
follows. First, for the polynomials z rb 1, replace terms corresponding to partitions with an odd 
number of odd parts by their negatives. Second, for polynomials invariant under ~ replace terms 
corresponding to partitions of odd size by their negatives. 



2.4 Limitations and Other Methods 

Cycle index techniques, while very useful, also have their limitations and are not always the best 
way to proceed, as the following examples demonstrate. 
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Example 1: Primitive prime divisor elements 

For integers b, e > 1 a primitive prime divisor of b e — 1 is a prime dividing b e — 1 but not 
dividing b % — 1 for any i with 1 < i < e. An element of GL(n, q) is called a primitive prime divisor 
(ppd) element if its order is divisible by a primitive prime divisor of q e — 1 with n/2 < e < n. The 
analysis in |NiP | derives elegant bounds on the proportions of ppd elements in the finite classical 
groups and applies them to the group recognition problem for classical groups over finite fields 
(determining when a group generated by a set of matrices contains SL(n,q)). We do not see how 
to get comparable bounds using generating function techniques. 

Example 2: Proportions of semisimple elements in exceptional groups 



Although Example 5 of Section 2.2 was estimating proportions of semisimple matrices, this was 
only for the finite classical groups, where the index n can take an infinite number of values. Cycle 
indices don't seem useful unless there is a tower of groups of varying rank available. 

Fortunately the computer package CHEVIE permits calculations precisely in finite rank cases 



such as the exceptional groups. Indeed this is how | GuLub | obtained estimates of the proportions 
of semisimple elements in the exceptional groups. 

Example 3: Non-uniform distributions on matrices 

The cycle indices give useful information about conjugacy class functions when the matrix is 
chosen uniformly at random. However there are other distributions on matrices which one could 
study and for which cycle index methods (at present) can not be applied. 

One example is random n x n matrices where the matrix entries are chosen independently 



according to a given probability distribution on F q . Charlap, Rees, and Robbins | ChReRc ] show 



that if the probability distribution is not concentrated on any proper affine subspace of F q , then 
as n — > oo the probability that the matrix is invertible is the same as for a uniform matrix. They 
use Moebius inversion on the lattice of subspaces of an n dimensional vector space and the Poisson 
summation formula. Is the same true for other natural conjugacy class functions? We expect 
that the answer is yes, which can be regarded as a type of "universality" result for the asymptotic 
description of random elements of GL(n,q) to be given in Subsection |3.1| . Analogous universality 
results are known for matrices with complex entries [3o|. For further information on the rank of 
random — 1 matrices, see [BKW] for sparse matrices, [Bo] for a survey of results on the rank over 



the real numbers, and also the discussion of work of Rudvalis and Shinoda in Subsection 3.2. 

It is conceivable that cycle index techniques will be able to handle certain natural non-uniform 
distributions on GL(n, q). This happens for the symmetric groups, where natural non-uniform mea- 



sures such as performing a g-riffle shuffle on a deck of cards has a useful cycle index [ DiaMcPi |,[ F10 | 



3 Running Example: General Linear Groups 

The purpose of this section is to give different ways of understanding the conjugacy class of a 
random element of GL(n, q). The analogous theory for other finite classical groups is mentioned in 
passing but is not treated in detail as many of the main ideas can be communicated using GL(n, q). 



Subsection 3.1 will show how this leads naturally to the study of certain probability measures 
M GL . U n on the set of all partitions of all natural numbers. Connections with symmetric function 
theory lead to several ways of growing random partitions distributed according to McL,u,q- One 
consequence is a motivated proof of the Roger s-Ramanuj an identities. 



3.1 Measures on Partitions 

The goal is to obtain a probabilistic description of the conjugacy class of a random element of 
GL(n,q). The ideas are based on |F1[. For this the following definition will be fundamental. 
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Definition: The measure Mql u ,g ori the set of all partitions of all natural numbers is defined 

by 



r=i q Q^ tK l> n 



(q)mi(X) 



The motivation for this definition will be clear from Theorem |||. The measure McL,u,q> while 
seemingly complicated, does have some nice combinatorial properties. For instance for partitions 
of a fixed size, this measure respects the dominance order on partitions (in this partial order A > fi 
if and only if Ai + • ■ • Aj > jXx + ■ ■ ■ + fii for all % > 1). In work with Bob Guralnick we actually 
needed this property. 

Lemma [| proves that for q > 1 and < u < 1, the measure McL,u,q is in fact a probability 
measure. There are at least three other proofs of this fact: an argument using q series, specializing 
an identity about Hall-Littlewood polynomials, or a slick argument using Markov chains and an 
identity of Cauchy. This third argument will be given in Subsection 3.4 . 

Lemma 4 If q > 1 and < u < 1, then McL,u,q defines a probability measure. 



Proof: Mci,u,q is clearly non-negative when q > 1 and < u < 1. Stong [ptl|l established an 
equation which is equivalent to the sought identity 

U \M 00 i 

V (A') 2 T-r 7l\ — II^i _ u)' 

As some effort is required to see this equivalence, we derive the identity directly using Stong's line 
of reasoning. 

First observe that unipotent elements of GL(n,q) corresponding to nilpotent n x n matrices 
(subtract the identity matrix), and that the number of nilpotent n x n matrices is g n ( n_1 ) by the 
Fine-Herstein theorem [ FeinHer| . The number of unipotent elements in GL(n,q) can be evaluated 
in another way using the cycle index of the general linear groups. Namely set x^^x = 1 \f (p = z — \ 
and set x<^a = otherwise. One concludes that 



Now multiply both sides by u n , sum in n, and apply Euler's identity 

y ^ = TT(— )■ 



□ 



The measure Mql u ^ q is a fundamental object for understanding the probability theory of con- 
jugacy classes of GL(n, q). This emerges from Theorem |j. 

Theorem 4 1. Fix u with < u < 1. Then choose a random natural number N with probability 
of getting n equal to (1 — u)u n . Choose a uniformly in GL(N,q). Then as (p varies, the 
random partitions X^,(a) are independent random variables, with A^ distributed according to 
the measure M GL u deg(<t,)^de g (<t,). 
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2. Choose a uniformly in GL(n,q). Then as n — > oo, the random partitions A^(a) converge in 
finite dimensional distribution to independent random variables, with A^ distributed according 
to the measure M GL 1 q d eg {4>) . 



Proof: Recall the cycle index factorization 



GL(n,q) 



U 



n=l 



n 

<$>+z 



U 



n-deg((p) 



1 + ^ ^ X0 ' A rr «) V(.Vi-'n , i , 



Setting all x^ t \ equal to 1 and using Lemma ||] shows that 

1 ^ ^ 1 



1 



<j>^Z r=l 



1 



M deg(4>) 
qr-deg(4>) 



Taking reciprocals and multiplying by the cycle index factorization shows that 



(1 



GL(n,q)\ 



U)U 



n=l 




GL,U de 9W ,g*9(0) (O) + X/ M GL ^ u deg{<t>) tq deg(<P) 

A:|A|>0 



(A)* 



</>,A 



This proves the first assertion of the theorem. For the second assertion, use Lemma [2] from Sub- 
section □ 



Remarks: 



1. Theorem || has an analog for the symmetric groups [ ShLl| . The statement is as follows. Fix u 
with < u < 1. Then choose a random natural number N with probability of getting n equal 
to (l — u)u n . Choose tt uniformly in Sn- Letting rii be the number of ' i- cycles of it, the random 
variables Hi are independent, with ni distributed as a Poisson with mean Furthermore if 
one chooses it uniformly in S n and lets n — ► oo, then the random variables rii are independent 
random variables, with n,, distributed as a Poisson(i). 

2. The idea of performing an auxiliary randomization of n is a mainstay of statistical mechanics, 
known as the grand canonical ensemble. For a clear discussion see Sections 1.7, 1.9, and 4.3 



of H 



3.2 Symmetric Function Theory and Sampling Algorithms 

The aim of this subsection is two-fold. First, the measures McL,u,q are connected with the Hall- 
Littlewood symmetric functions. Then we indicate how this connection can be exploited to give 
probabilistic methods for growing random partitions distributed as McL,u,q- The purpose is not to 
drown the reader in formulas, but rather to show that the connection between symmetric functions 
and probability is deep, beautiful, and useful in both directions. The results on this section are 
], except for the remark on how to make the algorithms terminate in finite 



based on [Fl] and 
time, which is joint with Mark Huber. 

To begin, we recall the Hall-Littlewood symmetric functions, which arise in many parts of math- 
ematics: enumeration of p groups, representation theory of GL{n, q), and counting automorphisms 



of modules. The basic references for Hall-Littlewood polynomials P\ is Chapter 3 of [Mac], which 
offers the following definition 



P\(x 1 , ■■■ ,x n ;t) 



T-T TT"li(A) \-t r 

lli>0llr=l ~\~t 



wes, 



n 

Kj 
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Here w is a permutation acting on the x- variables by sending X{ to x w uy Recall that mj(A) is the 
number of parts of A of size i. At first glance it is not obvious that these are polynomials, but the 
denominators cancel out after the symmetrization. The Hall-Littlewood polynomials interpolate 
between the Schur functions (t = 0) and the monomial symmetric functions (t = 1). 

Theorem m relates the measures MoLu,q to the Hall-Littlewood polynomials. Recall that n(X) = 

Ei(i-l)A< = E i (^). 
Theorem 5 



P f— 

Mgl,uM = U(1 



II I \\g: ' ' ' : q , 



Proof: From the above formula for Hall-Littlewood polynomials, it is clear that the only surviving 
term in the specialization P\(^, ^jz, • • • ; ^) is the term when w is the identity. The rest is a simple 
combinatorial verification. (Alternatively, one could use "principal specialization" formulas for 
Macdonald polynomials on page 337 of [ Mad] ) . □ 



Remark: The paper |F3| gives symmetric function theoretic generalizations of the measure 
M GL . u ^q on partitions. In the case of Schur functions s\, this measure depends on two infinite sets 
of variables Xj, m and assigns a partition A mass equal to s\(xi)s\(yi) riij(l — x iUj)- K is remarkable 
that precisely this measure arose in work of the random matrix community relating the distribution 
of the lengths of increasing subsequences of random permutations to the distribution of eigenvalues 
of random GUE matrices (these matrices have complex entries). To elaborate, the Robinson- 
Schensted-Knuth correspondence associates a random partition of size n to a random permutation 
of size n and the shape of the partition encodes information about the longest increasing subsequence 
of the permutation. Choosing the size of the symmetric group randomly (according to a Poisson 
distribution) gives a probability measure on the set of all partitions of all natural numbers which 
is a special case of the above Schur function measure. Then the coordinate change hj = X'i + Xj — j 
maps the set of row lengths {Xj} of the partition to a set of distinct integers {hj}. These hj 
can be viewed as positions of electrostatic charges repelling each other, and from this viewpoint 
the measure on subsets of the integers bears a striking resemblance to the eigenvalue density of 
a random GUE matrix. This fantastic heuristic can be made precise and led to a solution of the 
long-standing conjecture relating lengths of increasing subsequences of permutations to eigenvalues 
of random matrices. For these developments see [BOQl|, pc| and the many references therein. 



Now we return to the measure McL,u,q and describe an algorithm for growing random partitions 
according to this measure. 

The Young Tableau Algorithm 

Step Start with N = 1 and A the empty partition. Also start with a collection of coins indexed 
by the natural numbers, such that coin i has probability ^ of heads and probability 1 — j~ 
of tails. 

Step 1 Flip coin N. 

Step 2a If coin N comes up tails, leave A unchanged, set = + 1 and go to Step 1. 

Step 2b If coin N comes up heads, choose an integer S > according to the following rule. Set 

5 = 1 with probability 9 g jv_j~ ■ Set S = s > 1 with probability -rrzi — ~ — • Then 

increase the size of column s of A by 1 and go to Step 1. 
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As an example of the Young Tableau Algorithm, suppose we are at Step 1 with A equal to the 
following partition: 



□ □ □ □ 

□ □ 
□ 

Suppose also that N = 4 and that coin 4 had already come up heads once, at which time we added 
to column 1, giving A. We flip coin 4 again and get heads, going to Step 2b. We add a box to 
column 1 with probability q ^_ 1 , to column 2 with probability q q iZ_\ > to column 3 with probability 

q q i^i , to column 4 with probability 0, and to column 5 with probability q q i q 1 ■ We then return to 
Step 1. 

Theorem 6 For < u < 1 and q > 1, the Young Tableau Algorithm generates partitions which 
are distributed according to the measure McL,u,q- 

To give insight into the proof of Theorem |6|, we remark that it was deduced by proving a stronger 
result (Theorem]^) inductively and then taking the TV — > oo limit. As is clear from the statement of 
Theorem 0, the connection with Hall-Littlewood polynomials (in particular the ability to truncate 
them) was crucial. It is unlikely that the Young Tableau Algorithm would have been discovered 
without this connection. 

Theorem 7 Let P N (X) be the probability that the algorithm outputs A when coin N comes up tails. 
Then 

if X[ > N. 

Next we explain why the Young Tableau Algorithm is called that. A standard Young tableau 
T of size n is a partition of n with each box filled by one of {1, • • • , n} such that each of {1, • • • , n} 
appears exactly once and the numbers increase in each row and column of T. For instance, 



T 




3_ 


~2 




4 


8~ 




9~ 







is a standard Young tableau. Standard Young tableaux are important in combinatorics and rep- 
resentation theory. The Young Tableau Algorithm is so named because numbering the boxes in 
the order in which they are created gives a standard Young tableau. Thus although our initial 
interest was in the measure McL,u,q on partitions, the Young Tableau Algorithm yields more: a 
probability measure on standard Young tableaux. One consequence of this is a (new) representation 
of prinicipally specialized Hall-Littlewood polynomials as a sum of certain weights over standard 
Young tableaux. 

Let us indicate an application of this probability measure on standard Young tableaux. Rudvalis 
and Shinoda fRuShi studied the distribution of fixed vectors for the classical groups over finite 



fields. Let G = G(n) be a classical group (i.e. one of GL,U,Sp, or O) acting on an n dimensional 
vector space V over a finite field F q (in the unitary case F q 2) in its natural way. Let PG, n (k, q) be 
the chance that an element of G fixes a k dimensional subspace and let PG,oo(k,q) be the n — > oo 
limit of PG t n(k,q). They found (in a 76 page unpublished work) beautiful formulas for PG,oa(k,q)- 
Their formulas are (setting x = -): 
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i. ^L,oc(M) = [rr=i(i-z r )]7i 



(l-x) 2 -(l-x k ) 



k\2 



2. Pu,oo(k,q)= UZi 



1+x 2 



{l~x 2 )-{l-x 2k ) 



3. Ps Pt oo(k,q) — YlZl i+x r 



(l-x)---(l-x k ) 



4. Po,oo{k,q) = I! 



lr=0 l+x r 



x~^T 



(l-x)--(l-x k ) ' 



From a probabilistic perspective, it is very natural to try to interpret the factorizations in these 
formulas as certain random variables being independent (the paper |RuShi| gives no insight as 
to why these formulas have a product form). The Young tableau algorithm leads to such an 



understanding for the finite general linear and unitary groups; see [F3] for details. 
Remarks 



1. A skew diagram is the set theoretic difference between paritions /i,A with \x C A and a 
horizontal strip is a skew diagram with at most one square in each column. There is another 
algorithm for growing random partitions distributed according to McL,u,q in which one tosses 



coins and adds horizontal strips (as opposed to a box at a time). Details are in [F3|. 



2. (Joint with Mark Huber) We indicate how to make the Young Tableau Algorithm run on a 
computer, so as to terminate in finite time (clearly one can't flip infinitely many coins). Let 
a 7v be the number of times that coin N comes up heads; the idea is to first determine the 
random vector [a\ , a.2 , • • • ) and then grow the partitions as in Step 2b of the Young Tableau 
Algorithm. So let us explain how to determine (oi,02,---). For N > 1 let t^ N ' be the 
probability that all tosses of all coins numbered N or greater are tails. For N > 1 and j > 
let tj be the probability that some toss of a coin numbered N or greater is a head and that 

coin N comes up heads j times. It is simple to write down expressions for , , , • • ■ 
and clearly tW + Ej>o = L 

The basic operation a computer can perform is to produce a random variable U distributed 
uniformly in the interval [0, 1]. By dividing [0, 1] into intervals of length tg , ij , • • • and 
seeing where U is located, one arrives at the value of a\. Furthermore, if U landed in the 
interval of length then all coins come up tails and the algorithm is over. Otherwise, move 
on to coin 2, dividing [0, 1] into intervals of length , i ,tf , • • • and so on. 

For < u < 1 and q the size of a finite field, this algorithm terminates quickly. The 
probability of the algorithm stopping after the generation of the first uniform in [0, 1] is 
n^i(l - u/q % ) > n^i(l _ !/<?') > (1 _ l/<?) 2 > 1/4 where the second inequality is Corollary 
3.6 of gP2| . Should it be necessary to generate future uniforms, the same argument shows 
that the algorithm stops after each one with probalility at least 1/2. 



3.3 Sampling for a Given Size: Unipotent Elements 

An element of GL(n,q) is called unipotent if all of its eigenvalues are 1; a theorem of Steinberg 
asserts that the number of unipotent elements in GL(n, q) is g n ( ra_1 ) (this is the square of the order 
of a g-Sylow subgroup if q is prime). Unipotent elements are interesting because any element a in 
GL(n, q) can be written uniquely as the product a s a u where a s is semisimple and a u is unipotent. 
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Thus it is natural to study the random partition A^_i for unipotent elements in GL(n, q). This 
is the same as conditioning the measure McL,u,q to live on partitions of size n. This subsection 
explains how to modify the sampling method of Subsection |3,2j to sample from this conditioned 
version of McL,u,q and also from a (/-analog of Plancharel measure (related to the longest increasing 
subsequence problem). These results are joint with Mark Huber. 

Algorithm for Sampling from McL,u,q given that |A| = n 
Step Start with N = 1 and A the empty partition. 
Step 1 If n = then stop. Otherwise set h = 1 — 
Step 2 Flip a coin with probability of heads h. 

Step 2a If the toss of Step 2 came up tails, increase the value of iV by 1 and go to Step 2. 

Step 2b If the toss of Step 2 comes up heads, decrease the value of n by 1, increase A according 
to the rule of Step 2b of the Young Tableau Algorithm (which depends on N), and then go 
to Step 1. 

Theorem [8| will show that the above algorithm samples from Mgl,u.q conditioned to live on par- 
titions of size n. It is perhaps surprising that unlike the Young Tableau Algorithm, the probability 
of a coin coming up heads is independent of the coin number; it depends only on the number of 
future boxes needed to get a partition of size n. 

Lemma 5 Let Ni be the number of times that coin i comes up heads in the Young Tableau Algorithm 
with u = 1 and let N be the infinite vector with ith component N. 

n^ii^-T-) 

1. The probability that Ni = tii is — ~- 

2. 

1 1 



E 



Proof: The first assertion is clear. The second assertion is well known in the theory of partitions, 
but we argue probabilistically. Multiply both sides by II^i(l — ^0- Then note from the first 
assertion that the left hand side is the McL,i,q chance of having a partition of size a. Now use the 
second equation in the proof of Lemma ^| in Subsection . □ 



For Theorem || the notation Prob. is shorthand for the probability of an event. 

Theorem 8 The algorithm for sampling from McL,u.q conditioned to live on partitions on size n 
is valid. 

Proof: From the formula for McL,u,q, the conditioned measure for McL,u,q is the same as for 
MoLX,q- Now let 7ij be the number of times that coin i comes up heads in the Young Tableau 
Algorithm. Letting | denote conditioning, it suffices to show that 

Prob.(ni > 1| rij = s) = 1 -. 

3>i q 
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In fact (for reasons to be explained later) we compute a bit more, namely the conditional probability 
that m = a given that J2j>i n j = s - By definition this conditional probability is the ratio 

Prob.(m = a, Y.j>i n j = s) 
Prob.{Y,j>inj = s) 

The numerator and denominator are computed using Lemma [6| as follows: 



Prob.{rii = a,^nj = s) 



E 



n£i(i - w) 



E 



n^(i - w 



E 



ai- 



-a q is q£i>i aj 



n^a - W) 

q is q s - a {\)s-a ' 



Prob.^n, = s) = £ l /Q 

04+—= 

= E 



n~i(i - w) 



(<-l)«+Ej S1 (3-(<-l))«i 

n r °° =l (i - W) 



n r °° = ,(i - w) 

Thus Prob.(rii = 0| 2~]j>j n j = s ) = ^ an d the result follows. □ 



As mentioned in Subsection |3,2j there is a natural measure Mpi >q on the set of all partitions of all 
integers which when conditioned to live on partitions of a given size gives a ^-analog of Plancherel 
measure, which is related to longest increasing subsequence in non-uniform random permutations 
jFjJ. In what follows J a {q) is the polynomial discussed on pages 52-54 of fFlj , h(s) denotes the 
hook-length of a dot in A |Mac | and [n] = 9 ~^ is the g-analog of the number n. Recall that a skew 



diagram is the set theoretic difference between paritions [i, A with \i C A and that a horizontal strip 
is a skew diagram with at most one square in each column. 

Algorithm for Sampling from Mpi„ for q > 1 given that |A| = n 

Step Start with A the empty partition. 

Step 1 If n = then stop. Otherwise choose a with < a < n with probability 

1 n ( 1 ~ gn-q+i ) 2 • • • (1 ~ ^-) 2 J n -a{q) 
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Then increase A to A with probability 



^ qn(X) m6A(1 " ^ 



Q a q n(A) n s6 A(l-^iy) 

if A — A is a horizontal strip of size a and with probability otherwise. Finally replace n by 
n — a and repeat Step 1. 

Using Lemma ||, Theorem || proves that the algorithm for sampling from Mpi q conditioned to 
live on |A| = n works. We omit the details, which (given the background material in pT[| ) are 
analogous to the case of McL,u,q- 



Lemma 6 Let Ni be the number of times that coin i comes up heads in the algorithm from [FSJ 
for sampling from the measure Mpi tQ and let Ni be the infinite vector with ith component Ni. 

n^iLTw^-T) 

1. The probability that Ni = tii is — ^ . 3 — . 



1 J a {q) 



Theorem 9 The algorithm given for sampling from Mpi q with q > 1 conditioned to live on |A| = n 
is valid. 

3.4 Markov Chain Approach 

The main result in this subsection is a third method for understanding the measure McL,u,q prob- 
abilistically ( [|F7f |). The idea is to build up the random partition a column at a time; if the current 
column has size a, then the next column will have size b (with b < a) with probability K(a, b). The 
surprise is that this transition rule turns out to be independent of the columns, yielding a Markov 

2 

on the natural numbers. This Markov chain is diagonalizable with eigenvalues 1, ^, \ , • • •. It will 
be used to give a probabilistic proof of the Rogers- Ramanuj an identities in Subsection 

It is convenient to set Aq (the height of an imaginary zeroth column) equal to oo. For the entirety 



of this subsection, let P(a) be the McL,u,q probability that A' x = a. Theorem 10, which makes the 
connection with Markov chains, is proved in a completely elementary way. The argument reproves 
that McL,u,q is a probability measure (Lemma || of Subsection |3.1[) , shows that the asserted Markov 
transition probabilities add to one, and gives a formula for P(a). 

Theorem 10 Starting with X' = oo, define in succession A^A^,--- according to the rule that if 
A- = a, then X' i+1 = b with probability 



Then the resulting partition is distributed according to McL,u,q- 
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Proof: Suppose we know that Mgl,u,q is a probability measure and that 

«°(*)oo 



P(a) 



Q a { q )a{ q )a 



Then the McL,u,q probability of choosing a partition with A^ = r\ for all i is 

, Pro&.(A' = 00, A; = ri) Prob.(X' = 00, Aj = n, • • • , A^ +1 = r m ) 

10 °° J Pro6. (A' = 00) Prob.(X' = 00, A' x = n, • • • , A- = ri) ' 

Thus it is enough to prove the (surprising) assertion that 

Prob.(X' = 00, A' : = n, ■ ■ ■ , A-_ x = rj_i, A- = a, A- +1 = b) _ « 6 (^)a(f )a 

Pro6.(A^ = cx), A' x = n, • • • , AU = r^, \[ = a) " ^ 2 (I) a _ 6 (I) 6 (|) 6 ' 

for all i, a, 6, n, • • • , rj_i. One calculates that 



u riH hri_i 

E M GL,u,q{X) = , + .„ +rt P(a). 

A:A' 1 =T- 1 ,--,A^_ 1 =r i _ 1 y l < j7r , i-r 2 I, ? ;r i _ 2 -r i _i ^ ? M-i-a 

\'.= a 

Similarly, observe that 

? ,riH hn-i+a 

^ ^,u,q\ J r 2 + ... +r 2_ +a 2 1 1 1 J W 

A:A' 1 =r 1 ,--,A(_ 1 =r i _ 1 </ \ q )ri-r 2 \ q )ri- 2 -ri-i\ q )r i - 1 -a\ q ) a-b 

X'.=a,X'. , , =6 
I z+1 



Thus the ratio of these two expressions is 



^)a(§) £ 



as desired. Note that the transition probabilities must sum to 1 because 

EA=A' 1 =r 1 ,...,A;_ 1 =r i _ 1 MsL,U, 9 (A) 

E 



\'.=a,\ f . , , =b 



b<a 

A' 



for any measure McL,u,q on partitions. 

Thus to complete the proof, it must be shown that McL,u,q is a probability measure and that 



P , , 



Q a (q)a( q )a 

Since 
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it follows that 

V P(b)u a 

From this recursion and the fact that P(0) = (-)ooi one solves for P(a) inductively, finding that 

«°(f)oc 



P(o) 



a i^MV 



a 



Cauchy's identity (page 20 of [ Al|) gives that J2 a P( a ) = 1> so that McL,u,q is a probability measure. 
□ 

Theorem |ll| diagonalizes the transition matrix K, finding a basis of eigenvectors, which is 
fundamental for understanding the Markov chain (part 3 is stated as a Lemma in [ |A2fl ). Since the 
matrix K is upper triangular with distinct eigenvalues, this is straightforward. 

Theorem 11 1. Let C be the diagonal matrix with (i,i) entry ( q )i(¥)i- Let M be the matrix 



Then K = CMC , which reduces the problem of diagonalizing K to that of 



diagonalizing M . 

2. Let A be the matrix I --p- — 7-^ — I . Then the columns of A are eigenvectors of M for right 
multiplication, the jth column having eigenvalue 



3. The inverse matrix A 1 is 



(l-V^X-i)'-^)^,! 



Corollary § (immediate from Theorem will be useful for the proof of the Rogers- Ramanuj an 
identities in Section |3.5|. In the case L — > oo and j = 0, it is the so called Rogers- Selb erg identity. 



Corollary 2 Let E be the diagonal matrix with (i,i) entry ^ . Then K r = CAE r A 1 C 1 . More 
explicitly, 

Proof: This is immediate from Theorem |ll]. □ 
Remarks 

1. One of our motivations for seeking a Markov chain description of McL,u,q is work of Fristedt 
|Fris], who had a Markov chain approach for the measure P q on the set of all partitions of all 
natural numbers defined by P q (X) = ]T^i(l — <? l )g' A ' where q < 1. Fristedt's interest was in 
studying what a uniformly chosen partition of an integer looks like, and conditioning P q to 
live on partitions of size n gives a uniform partition. The measure P q is related to to vertex 
operators [01] and to the enumeration of ramified coverings of the torus [ Dij | . In this regard 
the papers [01] and [BIO] prove that the k point correlation function 
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A fc=li=l 

is a sum of determinants involving genus 1 theta functions and their derivatives and give 
connections with quasi-modular forms. It would be marvellous if the measure McL,u,q (being 
related to modular forms via the Rogers- Ramanuj an identities) is also related to enumerative 
questions in algebraic geometry. 

2. As mentioned in the introduction, the Markov chain approach gives a unified description of 
conjugacy classes of the finite classical groups. For the symplectic and orthogonal groups it is 
necessary to use two Markov chains K\ and K<i- For the symplectic case, steps with column 
number i odd use K\ and steps with column number i even use K2. For the orthogonal case, 
steps with column number i odd use Ki and steps with column number i even use K%. The 



Markov chains K\,Ki are the same for both cases! Details are in [F6|. The Markov chain 
approach is also related to quivers [F7|. 



3.5 Roger s-Ramanuj an Identities 



The Rogers- Ramanuj an identities [Re 



1 



„2 

q 



+ S (1 - ff)(l - q 2 ) ■ ■ ■ (1 - q n ) Ml (1 " 



7 n(n+l) 



+ S(l-9)(l-9 2 )-"(l 



n=l v 



.,5n-2 



5n-3^i 



are among the most interesting partition identities in number theory and combinatorics, with 
connections to Lie theory and statistical mechanics (see the discussions in | A2 | and [F7] for many 
references). One ongoing challenge in the subject (posed by Hardy) has been to find a proof of the 
Rogers-Ramanujan identities which is both motivated and simple. The purpose of this subsection is 
to describe such a proof (|F7|), which is also the first probabilistic proof of the Rogers-Ramanujan 
identities. 

To illustrate the idea we give the proof of the following generalization of the first Rogers- 
Ramanujan identity (called the Andrews-Gordon identity [ A3 ] , [ Gor| ] ) : 



E • 

ni,— ,rifc_i>0 



Nl- 



(V?)ni ■■■( 1 /q)n k _ 1 



n 



1 



r^O, ±k(mod 2k + l) 



1 - (i/qy 



where N% = rii H + 

The idea is simple. We study the distribution of the length of the first row of a random partition 
distributed as Mgl,i,<i- F rom the definition of McL,\,q the probability that the first row has length 
less than k is equal to 



nu 



1 



E 



1 



A:A' fc =0 q 



(Ai) 2 



- (A '*- )2 (!/<?) a; -a' 2 



Letting rij denote 



A^ +1 and iVj denote A^, this becomes 
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r=l 



-i>0 1 



which is a essentially the left hand side of the Andrews-Gordon identity. On the other hand the 
probability that the first row has length less than k is equal to the probability that the Markov 
chain of Section 3.4 is absorbed at at time k. Since we diagonalized the matrix associated to 



this Markov chain, it is straightforward to compute this probability. To get it into product form it 



is necessary to apply Jacobi's triple product identity which has a simple combinatorial proof | A1 | 



Further details are in [F7|. 

Next we argue that this proof is motivated. Certainly the measure Mqlx* is a natural object to 
study, given that it is the n —* oo limit law of A 2 _i for a random element of GL(n, q). It was natural 



to try to build up the random partitions A column by column as in Section |3.4j . Observing that the 
resulting Markov chain is absorbing at with probability one, the time to absorption (equivalent to 
the distribution of the length of the first row) is the most natural quantity one could examine. The 
final step is applying Jacobi's triple product identity, and thus going from a "sum = sum" identity to 
a "sum = product" identity. As mentioned above Jacobi's triple product identity is easy to verify, 
but one still wants a motivation for trying to write the left hand side of the Andrews-Gordon 
identity in product form. One motivation is Baxter's work on statistical mechanics (surveyed in 
|A2| , [ Baxl | , [ Bax^ j ) in which he really needed "sum = product" identities and was led to conjecture 
analogs of Rogers-Ramanujan type identities. Although a proof of the Rogers-Ramanujan identities 
doesn't emerge from his work, it is clearly one of the truly great accomplishments in mathematics 



and his book [Baxl] has been very influential. A second motivation is our work on the n — > oo 
asymptotic probability that an element of GL(n, q) is semisimple. The argument, recorded in 
[Fl] or the more readily available [F4] needed a "sum = product" identity. The corresponding 



computation in [F9] for the finite afflne groups needed both Rogers-Ramanujan identities. 

Andrews' paper [A.4] notes that many proofs of the Rogers-Ramanujan identities make use of 
the following result called Bailey's Lemma, alluded to in [Bai] and stated explicity in [AS]. A pair 
of sequences {o^} and are called a Bailey pair if 



o , 



<r(V?)i 



-f3 r , then {a' L } and {P' L } are 



( 1 /q)L-r(u/q) L+r 

Bailey's Lemma states that if a' L = and 0' L = J2r=o 

a Bailey pair. From the viewpoint of Markov chains, this case of Bailey's Lemma is clear. To 
explain, let A,D,M be as in Theorem 11 (recall that M = ADA^ 1 ). Viewing a = d*L and f3 = Pl 
as column vectors, the notion of a Bailey pair means that (3 = Aa. This case of Bailey's Lemma 
follows because 

3' M I ADA' 1 (3 = ADa = Aa'. 



As Andrews explains in [A.2], the power of Bailey's lemma lies in its ability to be iterated and gives 
a short proof of the Rogers-Selberg identity (Corollary |2| in Section 3.4). From the remarks in this 
paragraph it is clear that iterating Bailey's lemma corresponds to taking several according to the 
Markov chain K. This demystifies the Bailey's Lemma proofs of the Rogers-Ramanujan identities, 
which strike this author as unmotivated. The fact that the Markov chain approach has analogs for 
other finite classical groups and for quivers is further evidence of its naturality. 



26 



4 Upper Triangular Matrices 



This section surveys probabilistic aspects of conjugacy classes in the group T(n, q) of upper tri- 
angular matrices over finite fields with l's along the main diagonal. At present little is known 
about conjugacy in T(n,q). 



The papers |VAr 



| VAr V | study the number of conjugacy classes. 
Kirillov [Kir] calls for an extension of his method of coadjoint orbits for groups over real, complex, 
or p-adic fields to the group T(n, q) and gives premilinary connections with statistical physics; the 



paper [IsKar| gives a counterexample to one of his conjectures. As we do not see how to further 
develop those results or improve on their exposition, we instead focus on a simpler problem: the 
probabilistic study of Jordan form of elements of T(n,q). 



Subsection 4.1 describes a probabilistic growth algorithm for the Jordan form of upper triangular 
matrices over a finite field. This is linked with symmetric function theory and potential theory on 



Bratteli diagrams in Subsection 4.2. 



4.1 Growth Algorithm for Jordan Form 

Theorem [l2| gives a probabilistic growth algorithm for the Jordan form of random elements of 
T(n, q). Its proof uses elementary reasoning from linear algebra. 

Theorem 12 ( ^Ki^ j,^j) The Jordan form of a uniformly chosen element ofT(n,q) can be sampled 
from by stopping the following procedure after n steps: 

Starting with the empty partition, at each step transition from a partition X to a partition A by 
adding a box to column i chosen according to the rules 



i = 1 with probability 



i = j > 1 with probability 



l 

A' 
q 3 



J-l 



Theorem [l2| leads to the following central limit theorem about the asymptotic Jordan form of 
an element of T(n,q). 



Theorem 13 (j^) Let A be the partition corresponding to the Jordan form of a random element of 
T(n,q 
Then 



T(n,q). Let Prob n denote probability under the uniform measure on T(n,q) and let pi = —= 



lim n ^ 00 Prob n '{— — < Xi, i = 1, • • • , k) = (2ir)~ a 



n 



e 2 



h<Qt,t> 



dt 



for any (xi, ■ ■ ■ ,Xk) G R k , where the covariance matrix equals 

Q = diagipx,- ■ ■ ,p k ) - {piPj)Xj=v 

4.2 Symmetric Functions and Potential Theory 

Given the usefulness of symmetric functions in the probabilistic study of the measure Mgl,u,q, it 
is natural to seek an analogous understanding of Theorem [l^. That is the topic of the present 



subsection. The ideas here are from the report |F8|. 

The first step is to link the probability that an element of T(n, q) has Jordan form of type A 
with symmetric function theory. For the rest of this section, P\(q,t) denotes a Macdonald poly- 
nomial, K^\(q,t) denotes a Kostka-Foulkes polynomial, and f^ is the dimension of the irreducible 
representation of S n corresponding to the partition fi (see [ Mac ] for background). Note that when 
q = the Macdonald polynomial is our friend, a Hall-Littlewood polynomial. 
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Theorem 14 (Ww) The probability that a random element ofT(n,q) has Jordan form of type A 



is 

Pa(i - -, - - \, ■ ■ • ; o, -) £ / m ^m(o, 

9 9 9 2 9 fa 

Next we give some background on potential theory on Bratteli diagrams. This is a beautiful 
subject, with connections to probability and representation theory. We recommend [Kel] for back- 



ground on potential theory with many examples and [BO]] for a survey of recent developments. 
The basic set-up is as follows. One starts with a Bratteli diagram; that is an oriented graded graph 
r = U n >or n such that 

1. To is a single vertex 0. 

2. If the starting vertex of an edge is in Tj, then its end vertex is in Tj+i. 

3. Every vertex has at least one outgoing edge. 

4. All Ti are finite. 

For two vertices A, A 6 T, one writes A f A if there is an edge from A to A. Part of the 
underlying data is a multiplicity function k(X, A). Letting the weight of a path in T be the product 
of the multiplicities of its edges, one defines the dimension dim(A) of a vertex A to be the sum of 
the weights over all maximal length paths from to A (this definition clearly extend to intervals). 
Given a Bratteli diagram with a multiplicity function, one calls a function harmonic if 0(0) = 1, 
0(A) > for all A G T, and 

0(A) = £ K (A,A)0(A). 

A:A/A 

An equivalent concept is that of coherent probability distributions. Namely a set {M n } of proba- 
bility distributions M n on T n is called coherent if 

*m(AMA,A) 

A:t^A 

The formula allowing one to move between the definitions is 0(A) = . 

One reason the set-up is interesting from the viewpoint of probability theory is the fact that 
every harmonic function can be written as a Poisson integral over the set of extreme harmonic 
functions (which is often the Martin boundary). For the Pascal lattice (vertices of T n are pairs 
(k, n) with k = 0, 1, • • • , n and (k, n) is connected to (k, n + 1) and (k + 1, n + 1)), this fact is the 
simplest instance of de Finetti's theorem. When the multiplicity function k is integer valued, one 
can define a sequence of algebras A n associated to the Bratteli diagram, and harmonic functions 
correspond to certain characters of the inductive limit of the algebras A n . 

Next we define a branching for which the probability that an element of T(n, q) has Jordan 
type A is a harmonic function. First some notation is needed. For A /* A, let Ra/x (resp. C^/x) 
be the boxes of A in the same row (resp. column) as the boxes removed from A to get A. This 
notation differs from that in [Mad] . Let a\(s), l\(s) be the number of dots in A strictly to the east 
and south of s, and let h\(s) = a\(s) + l\(s) + 1. 

Definition 1: For < q < 1 and < t < 1, the underlying Bratteli diag ram T has as level r n 
all partitions A of n. Letting i be the column number of the dot removed to go from A to A, for 
A y A, define the multiplicty function as 
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I I — g«A(s)+l^A(s) i _ ^a A (s)^A(s)+l 

K ( X > A ) = ^i [] 1 _ q a x (s)+l t h(s) II l _ q a x (s) t h(s)+l ■ 



Equation 1. 10 of [GarsH| proves that 



^m(A) = ^ xy ^/^ M (g,t). 

Definition 2: For 0<g<l,0<t<l and < x±,X2, ■ ■ ■ such that ^ Xj = 1, define a family 
{M n } of probability measures on partitions of size n by 



M n (A) 



(1 - g)l A lP A (x; g, t) /^^ M (g, *) 

lL eA (l - g«A(«)+itJA(-)) 
(l - g)l A lp A (x;g,f)H A )dim(A) 

lL GA (l - q a ^ +l t 1 ^) 

Consider the specialization that q = and t = |, where this second g is the size of a finite 



field. Further, set X{ = — rr — Then Theorem 14 implies that M n (A) is the probability that a 



uniformly chosen element of T(n, g) has Jordan type A. The multiplicities have a simple description; 
letting i be the column to which one adds in order to go from A to A, it follows that k(A, A) = 
qK _|_ qK~ x -\- ■ ■ ■ + q X i+i. Second, dim(A) reduces to a Green's polynomial Q A (q) = Qf 1 n\(q) as in 



Section 3.7 of [ Mac ], These polynomials are important in the representation theory of the finite 
general linear groups. This specialization was the motivation for Definition 2. 
The connection with potential theory is given by the following result. 



Theorem 15 ([F8J) The measures of Definition 2 are harmonic with respect to the branching of 
Definition 1. 

It is elementary and well-known that if one starts at the empty partition and transitions from 
A to A with probability K ^M^^(^lm(A)^ ' one S e ^ s samples from any coherent family of measures 
{M n }. Applying this principle to the above specialization in which M n (A) is T(n,g) and using 



Macdonald's principal specialization formula (page 337 of |Mac|) gives the advertised proof of 
Theorem [l2| by means of symmetric functions and potential theory. 
Remarks: 



1. As indicated in [Fa] , the example of Schur functions (q = t < 1) is also interesting. The 
measure M n (A) reduces to s A / A , where s A is a Schur function. Setting x\ = ■ ■ ■ = x n = - and 
letting n — > oo, one obtains Plancherel measure, which is important in representation theory 
and random matrix theory. Letting x\ = • • • = x n satisfy ^ Xj = 1 (all other Xj = 0) gives 



a natural deformation of Plancherel measure, studied for instance by [ItTWi]. Stanley [Sta] 
shows that this measure on partitions also arises by applying the Robinson-Schensted-Knuth 
algorithm to a random permutation distributed after a biased riffle shuffle (in other words, 
this measure encodes information about the longest increasing subsequences of permutations 
distributed as shuffles). 
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2. It has been pointed out to the author that the branchings k(X, A) of Definition 1 are related 
to the branchings r(A, A) of [Ke2] by the formula 



k(A,A) = /(A)t(A,A)/(A) 



-i 



for a certain positive function /(A) on the set of vertices, which implies by [Ke3] that the 
boundaries of these two branchings are homeomorphic and that the branchings of Definition 
1 are multiplicative. Kerov |Ke2| ] has a conjectural description of the boundary. It has been 
verified for Schur functions M, Kingman branching [Kin], and Jack polynomials [KcOOl], 



but remains open for the general case of Macdonald polynomials. In particular, it is open 
for Hall-Littlewood polynomials, the case related to T(n,q). It is interesting that the k(A, A) 
of Definition 1 are integers for Hall-Littlewood polynomials, whereas the r(A, A) of [ Ke2| are 
not. 
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